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In the United States Patent and Trademark Office 



TO ALL WHOM IT MAY CONCERN: 

BE IT KNOWN, that we: David A. Glanzer, a citizen of the 
United States and resident of Georgetown, Texas; Lee A. Neitzel, 
a citizen of the United States and a resident of Austin, Texas; 
Raymond D. Sawyer, a citizen of the United States and a resident 
of Raynham, Massachusetts; Tony J. Kozlik and Colin R. Corles, 
citizens of the United States and residents of Phoenix, Arizona; 
Stephen K. Brodman, a citizen of the United States and a resident 
of Needham, Massachusetts; Johan L. Tegnell, a citizen of Sweden 
and resident of Mansfield, Massachusetts; William M. Hawkins, a 
citizen of the United States and a resident of Bloomington, 
Minnesota; and Michael D. Hirst, a citizen of the United States 
and a resident of Lakeville, Massachusetts have invented certain 
new and useful improvements in 

BLOCK-ORIENTED CONTROL SYSTEM ON HIGH SPEED ETHERNET, 

of which the following is a specification. 
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SPECIFICATION 



CROSS-REFERENCE TO RELATED APPLICATIONS 

This application is a continuation-in-part (CIP) application 
of U.S. Application No. 08/916,178 filed August 21, 1997, and is 
5 related to and claims priority from U.S. Provisional Application 
No. 60/024,346 filed August 23, 1996, and U.S. Provisional 
Application No. 60/139,814 filed June 21, 1999, all of which are 
hereby incorporated by reference in their entirety. 

10 TECHNICAL FIELD 

The present invention relates to control system 
architecture. More particularly, the present invention relates to 
an open, interoperable distributed control system in a high 
15 performance network environment. 



BACKGROUND OF THE INVENTION 



m 
& 
m 

m 

*■ 20 Automatic control systems are critical to all- sectors of 

j~ ' industry such as process control, discrete control, batch control 
^ (process and discrete combined) , machine tool control, motion 

a control, and robotics. One of the strongest needs in modern 

O control systems is development and use of "open" and 

Cft 25 "interoperable" systems. Open, interoperable systems allow 
fU control devices made by different manufacturers to communicate 

M 1 and work together in the same system without the need for custom 

0: programming. "Fieldbus" is the common term used to describe 

Q; these types of control systems. 

30 The movement toward open, interoperable fieldbus systems is 

driven by device manufacturers and end users. Manufacturers want 
open, interoperable systems because it allows them to sell their 
products to more end users while reducing development costs. End 
users want open, interoperable systems so that they can select 
35 the best control devices for their system regardless of the 
device manufacturer . 

There has also been a trend toward distribution of control 
functions into intelligent devices. In centralized control 
systems, a central controller performs all the control functions. 
40 In distributed control systems, more than one control device 

operating in the system takes an active role in the control 
functions. Although both centralized and decentralized systems 
use a communication network, decentralized systems reduce overall 
system costs by reducing or eliminating the centralized 
■ 45 controller functions between the control devices and the human- 
machine interface . 
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In order for distributed control systems to be truly open 
and interoperable, both the communications system and the user 
layer (above the communication system layers) must be specified 
and made open. One of the truly open and interoperable 
distributed systems is the fieldbus system provided by the 
Fieldbus Foundation. The Foundation™ fieldbus user layer is 
described, e.g., in US Patent Application 08/916,178 (hereafter 
the "'178" application) filed August 21, 1997, entitled "BLOCK- 
ORIENTED CONTROL SYSTEM", and assigned to the assignee of the 
present application . 

The lower speed 31.25 kilobits per second fieldbus (HI) used 
by the Foundation™ fieldbus is described in part by International 
Electrotechnical Committee (IEC) Standard IEC 61158, the entirety 
of which is hereby incorporated by reference herein. 

While the Foundation™ fieldbus provides the open and 
interoperable solution for the Hi control capability, there is a 
great need to provide an open and interoperable solution for 
distributed control on a very high performance communication 
system typically called a fieldbus "backbone" network. The 
backbone network aggregates information from the lower speed 
control devices, e.g., the HI and other control devices, which is 
used in supervisory and advanced control applications. The 
backbone is also needed for integration of control information 
into the enterprise's Management Information Systems (MIS). 

One of the widely accepted standards for high performance 
communications signaling is Ethernet. Invented by Xerox in the 
1970' s, Ethernet has progressed from an initial speed of 10 
Megabits per second, to 100 Megabits per second, to 1 Gigabit per 
second and beyond. Ethernet signaling is specified in an 
Institute of Electrical and Electronics Engineers (IEEE) standard 
(IEEE 802.3). Ethernet signaling is the underlying technology 
used by the Internet. The Internet protocols are specified by 
the Internet Engineering Task Force (IETF) and are issued as 
Request For Comment (RFC) specifications. 

Although Ethernet /Internet technology provides the basic 
services for a high performance fieldbus backbone, it does not 
provide for all of the functions needed for use in distributed 
control systems. In particular, IEEE and IETF do not have 
suitable open and interoperable solutions for integration of 
distributed control systems (e.g., the HI subsystem), system time 
synchronization, and fault tolerance. 

The method of transferring information from lower speed 
fieldbuses to the Ethernet used by organizations such as Open 
DeviceNet™ Vendor Association, Inc., ("EtherNet/IP ') and 
PROFIBUS International, ("FKOFINet") are not suitable for use in 
the high performance environment because they encapsulate the 
lower speed protocol packets in an Ethernet frame. This method, 
known as "tunneling," is common in centralized control systems, 
but is inadequate for high performance distributed control 
systems. Although simpler to specify, tunneling would require 
too many Transport Control Protocol (TCP) connections with the 
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resulting interrupt processing and memory overhead on the devices 
connected to the fieldbus backbone. In addition tunneling wastes 
much of the Ethernet bandwidth because the lower speed protocol 
packets (e.g., the HI packets) are small and in many cases the 
Ethernet packet overhead would be bigger than a lower speed 
protocol packet. 

Devices connected to the Ethernet must have a common sense 
of system time for time stamp and function block scheduling 
(control) purposes. For high performance distributed control, 
system time often needs to be accurate to within less than 1 
millisecond. Heretofore, there is no known solution that 
provides this accuracy using the Commercial Off The Shelf (COTS) 
Ethernet equipment . 

Fault tolerance of the Ethernet communication media and 
devices connected to the Ethernet is required for high 
performance distributed control applications. There is no known 
solution that provides the required fault tolerance using 
standard COTS Ethernet equipment. All of the prior attempts in 
providing the required fault tolerance require special 
Ethernet/Internet electronic hardware and/or software, and/or a 
non-standard "redundancy manager" device to be added to the 
Ethernet . 

Thus, what is needed is an open, interoperable solution 
optimized for integration of distributed control systems and 
other control devices in a high performance fieldbus backbone. 

What is also needed is an open, interoperable solution that 
provides system time synchronization suitable for distributed 
control applications operable over a high performance fieldbus 
backbone . 

What is also needed is an open, interoperable solution that 
provides a fault tolerant high performance fieldbus backbone as 
well as fault tolerant devices that are connected to the fieldbus 
backbone . 

SUMMARY OF THE INVENTION 

The present invention overcomes the shortcomings described 
above and provides a new and improved distributed control system, 
which operates on a high performance backbone, e.g., the standard 
COTS Ethernet and Internet technology. The embodiments of the 
present invention are collectively referred to herein as the 
"High Speed Ethernet" (HSE) . HSE includes the features of the 
distributed control system described by the x 178 application and 
Foundation™ fieldbus specifications (which are listed in Appendix 
A as the Reference Set 1), and further includes three new 
protocols described in the supporting specifications thereof, 
which are listed in Appendix A as the Reference Set 2. In 
particular, the new protocols are referred to herein as: the HSE 
Field Device Access (FDA) Agent, the HSE System Management Kernel 
(SMK) , and the HSE Local Area Network Redundancy Entity (LRE) . 

The HSE FDA Agent allows System Management (SM) and Fieldbus 
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Message Specification (FMS) services used by the HI devices to be 
conveyed over the Ethernet using standard Internet User Data 
Protocol (UDP) and Transport Control Protocol (TCP) . This allows 
HSE Devices on the Ethernet to communicate to HI devices that are 
5 connected via a "HSE Linking Device." The HSE FDA Agent is also 
used by the local Function Block Application Process (FBAP) in a 
HSE Device or HSE Linking Device. Thus, the HSE FDA Agent 
enables remote applications to access HSE Devices and/or HI 
devices through a common interface. 
10 The HSE SMK ensures that system level functions in each 

device are coordinated. These functions include system time, 
addition and removal of devices from the network, and function 
block scheduling. HSE SMK uses local clock that operates to keep 
a local time, and keeps the difference between the local time and 
15 a system time provided by a time server within a value specified 
by the time sync class (See Reference Set 1 of Appendix A 
herein) . The local time is used to time stamp events so that 
event messages from devices may be correlated across the system. 
Local time is also used to schedule the execution of the local 
20 function blocks. 

HSE fault tolerance is achieved by operational transparency 
i.e., the redundancy operations are not visible to the HSE 
applications. This is necessary because HSE applications are 
required to coexist with standard MIS applications. The HSE LRE 
25 coordinates the redundancy function. Each HSE Device 

^3 periodically transmits a diagnostic message representing its view 

''-}■ of the network to the other HSE Devices on its Ethernet 

=' interfaces (commonly called Ethernet "Ports") . Each device uses 

Q the diagnostic messages to maintain a Network Status Table (NST) , 

P 30 which is used for fault detection and Ethernet transmission port 
ID; selection. There is no central "Redundancy Manager' 7 . Instead, 

each device determines how it should behave in response to faults 
it detects . 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Features and advantages of the present invention will become 
apparent to those skilled in the art from the following 
description with reference to the drawings, in which: 

Figure 1 is a block diagram showing an exemplary embodiment 
of a high performance distributed control system in accordance 
with the principles of the present invention; 

Figure 2 is a block diagram showing an exemplary embodiment 
of device system architecture of a high performance distributed 
control system in accordance with the principles of the present 
invention; 

Figure 3 is a block diagram showing an exemplary embodiment 
of the structure of the High Speed Ethernet Management 
Information Base of the device system architecture shown in 
Figure 2; 

Figure 4 is a block diagram showing an exemplary embodiment 
of the device system architecture shown in Figure 2, showing the 
various local interfaces of the High Speed Ethernet Field Device 
Access agent; 

Figure 5 is a block diagram showing an exemplary embodiment 
of the relevant portions of the high performance distributed 
control system involved in time synchronization process in 
accordance with the principles of the present inventions- 
Figure 6 is a flow diagram illustrative of an exemplary 
embodiment of the process of time synchronization in accordance 
with an embodiment of the principles of the present inventions- 
Figure 7A is a timing diagram illustrative of a starting 
time offset before the time synchronization process in accordance 
with an embodiment of the principles of the present inventions- 
Figure 7B is a timing diagram illustrative of a starting 
time offset after the time synchronization process in accordance 
with an embodiment of the principles of the present invention; 
and 

Figure 8 is a block diagram showing an exemplary embodiment 
of the redundant topology of a high performance distributed 
control system in accordance with the principles of the present 
invention . 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

For simplicity and illustrative purposes, the principles of 
the present invention are described by referring mainly to 
exemplary embodiments, particularly, with specific exemplary 
implementations of distributed control system in an Ethernet 
network. However, one of ordinary skill in the art would readily 
recognize that the same principles are equally applicable to, and 
can be implemented in, other implementations and designs using 
any other high speed networks, and that any such variation would 
be within such modifications that do not depart from the true 
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spirit and scope of the present invention. 

A: HSE DISTRIBUTED CONTROL SYSTEM OVERVIEW 



Referring to FIG. 1, an example of a high performance 
control system 100 is shown where standard COTS Ethernet 
equipment 130 is used to interconnect HSE Linking Devices 110 and 
HSE Devices 120 to an Ethernet Network 140. The HSE Linking 
Devices 110 in turn connect to HI Devices 170 using HI Networks 
150. Other types of equipment such as a Personal Computer (PC) 
160 may also be connected to the Ethernet Network 140. 

The actual Ethernet network topology and COTS Ethernet 
equipment configuration will depend on the particular application 
needs. However any Ethernet network topology or configuration 
using standard COTS Ethernet equipment other than the exemplary 
topology shown in Fig. 1 may be used, and such variations would 
be within such modifications that do not depart from the true 
spirit and scope of the present invention. 



The HSE system architecture in accordance with an embodiment 
of the principles of the present invention is shown in FIG. 2. 
The HSE system architecture is designed to meet the functional 
needs of the high performance distributed manufacturing and 
process control environments, e.g., in a high speed Ethernet 
network. It permits distributed automation systems to be 
constructed from various control and measurement devices 
manufactured by different vendors. The HSE system architecture 
is described by architecture components that have been adapted to 
the specifics of both the HI and HSE environments. 

The various protocols and standards referenced in the 
following disclosure are described in detail in the manuals and 
specifications listed in Appendix A herein, which are available 
from the Fieldbus Foundation, a not-for-profit organization 
headquartered in Austin, Texas, and the respective current 
versions as of the filing date of the present invention of all of 
which are hereby incorporated by reference in their entirety 
herein. Each of the architecture components of the HSE system 
architecture (shown in Fig. 2) will now be described in more 
detail . 



Application Process (AP) is a term defined by the 
International Standards Organization (ISO) Open Systems 
Interconnect (OSI) Reference Model (RM) , ISO 7498, to describe 
the portion of a distributed application that is resident in a 
single device. The term is used in the following description to 
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A.l: HSE SYSTEM ARCHITECTURE 



A. 2: Function Block Application Process 
Virtual Field Device (FBAP VFD) 




refer to the entities within a device that performs a related set 
of functions, such as function block processing, network 
management, and system management. 

Virtual Field Device (VFD) is a term defined by the Fieldbus 
Foundation (See Fieldbus Message Specification FF-870 listed in 
Reference Set 1 in Appendix A herein) . A VFD makes the 
parameters of an AP visible to a communication network. 

In accordance with the principles of the present invention, 
the HSE system architecture (shown in Fig. 2) supports the 
Function Block Application Process Virtual Field Device (FBAP 
VFD) 260. The FBAP VFD 260 provides a common means for defining 
inputs, outputs, algorithms, control variables, and behavior of 
the automation system. There may be multiple FBAP VFDs 260, 
e.g., n FBAP VFDs as shown, in a device in order to satisfy the 
particular needs an application. The FBAP VFD may or may not be 
present in an HSE Device or HSE Linking Device. If the HSE FBAP 
VFD is present, the device is sometimes also referred to as a 
"HSE Field Device." In the following descriptions, however, the 
FBAP VFD is to be assumed to be present in the HSE Device and HSE 
Linking Device, even if the term "HSE Field Device" is not used. 

A standard set of function block classes and parameters are 
defined by the Fieldbus Foundation, e.g., in one or more of the 
specifications listed in Appendix A herein. Manufacturers of 
control devices may append their own parameters to the standard 
set of parameters to accommodate additional function block 
definitions as new requirements are discovered, and as technology 
advances. A more detailed description of the function block 
classes and parameters may be found, e.g., in Function Block 
Application Process-Part 1 Specification FF-890 listed in 
Reference Set 1 of Appendix A herein. 

A. 3: HI Interface 

Each HI Network 150 attached to a HSE Linking Device 110 
(shown in FIG. 1) requires a HI Interface 240. The Bridge 250 is 
used to convey HI Network messages directly between other HI 
Interfaces 240 within the same HSE Linking Device 110 (shown in 
FIG. 1). A HSE Linking Device may comprise, e.g., a HSE Device 
120 (shown in FIG. 1) that includes at least one HI Interface 
240. 

A more detailed description of a HI Interface may be found 
in the Fieldbus Message Specification FF-870, Fieldbus Access 
Sublayer Specification FF-821, Data Link Services and Data Link 
Protocol Specifications FF-821, 822, and Data Link Protocol 
Specification for Bridge Operation Addendum FF-806, all of which 
are listed in the Reference Set 1 of Appendix A herein. 

A. 4: Ethernet/ Internet Stack 

The HSE system architecture uses a standard COTS 
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Ethernet /Internet ("stack") 280 for communication with other 
devices on the Ethernet Network 140. The Ethernet/Internet stack 
used by HSE consists of Distributed Host Control Protocol (DHCP) 
285, Simple Network Time Protocol (SNTP) 286, and Simple Network 
5 Management Protocol (SNMP) 287, which in turn use Transport 
Control Protocol (TCP) 283 and User Data Protocol (UDP) 284 
services . 

TCP 283 and UDP 284 in turn use the standard Internet 
Protocol (IP) 282 services, which uses the standard IEEE Ethernet 
10 802.3 Media Access Control (MAC) and Physical (PHY) Layers 281. 
The PHY layer in 281 connects to one or more Ethernet Networks 
140. 

The Internet DHCP, SNTP, SNMP, TCP, UDP and IP protocols are 
specified by the Internet Engineering Task Force (IETF) Request 
15 For Comment (RFC) specifications. The IETF RFCs are listed in 
Appendix B herein, which are hereby incorporated by reference 
herein in their entireties. An Institute of Electrical and 
Electronics Engineers (IEEE) standard (IEEE 802.3), the entirety 
Q of which is hereby incorporated by reference herein, describe the 

y3 2 0 Ethernet MAC and PHY layers. The specific use of each layer and 
lit protocol are detailed in the Ethernet Presence Specification FF- 

yj 586 listed in Reference Set 2 of Appendix A herein. 

(K3i By preserving the standard use of the Ethernet/Internet 

p stack, the HSE system architecture ensures interoperability among 

y} 25 the different stack manufacturers. 

H 

=, A. 5: HSE Management Agent 

0 

(p Again referring to FIG. 2, in general, the HSE Management 

fij: Agent 270 uses DHCP 285 for acquiring an IP address for the 

U 30 device, SNTP 286 for maintaining time synchronization with a time 
Q server, and SNMP 287 for managing the TCP, UDP, and IP protocol 

Q; layers. HSE Management Agent use of DHCP, SNTP and SNMP is 

routine and complies with standard practices known to those 
familiar with the Internet protocols, e.g., according to IEEE 
35 802.3. 

The HSE Management Agent uses SNMP 287 for managing the 
Internet layer protocols. Specifically, the HSE Management Agent 
270 provides Ethernet Network access to the standard Management 
Information Base-II (MIB II) as defined by SNMPv2 in RFC 1213 and 

40 RFC 1643 (see Appendix B) , and as defined also by Ethernet 
Presence FF-586 listed in Reference Set 2 of Appendix A herein. 

In accordance with an embodiment of the present invention, 
in order to comply with the ISO standards, the HSE Management 
Information Base (HSE MIB) 271 comprises of a standard part, 

45 which is the second version of MIB-II as defined in RFC 1213 and 
a HSE specific part (which is defined under the private 
enterprises level) . For convenience in understanding, the 
detailed structure of the HSE MIB 271 is shown in FIG. 3. The 
standardized structure of the HSE MIB 271 provides a profile 
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allowing interoperability, making the device appear as a well- 
behaved node . 



Referring again to FIG. 2, the HSE core portion 200 of the 
HSE system architecture identifies the new HSE capability in 
accordance with the principles of the present invention. The HSE 
core 200 provides the essential capabilities and integration 
needed to realize the high performance distributed control using 
HSE Devices, HSE Linking Devices and standard COTS Ethernet 
equipment . 



The HSE System Architecture includes a Network Management 
Agent VFD (NMA VFD) 210 for each HSE Device and each HSE Linking 
Device. The NMA VFD provides means for configuring, controlling 
and monitoring HSE Device and HSE Linking Device operation from 
the network. 

Management information is contained in the Network 
Management Information Base (NMIB) 213 and the System Management 
Information Base (SMIB) 212. Using the configuration management 
capabilities of the NMA VFD, parameters are set in the NMIB and 
SMIB to support data exchanges with other devices in the system. 
This process involves defining the transfers between devices and 
then selecting the desired communications characteristics to 
support the transfers. 

The NMA VFD can also be configured to collect performance 
and fault related information for selected transfers. This 
information is accessible during run-time, making it possible to 
view and analyze the behavior of device communications. If a 
problem is detected, performance is to be optimized, or device 
communications are to be changed, then reconfiguration can be 
performed dynamically while the device is still operating. 

NMA VFD parameters and behavior are further defined in the 
HSE Network Management Specification FF-803 listed in the 
Reference Set 2 of Appendix A herein. 



The HSE Field Device Access (FDA) Agent will now be 
described with References to FIG. 4, which is the same figure as 
FIG. 2 except that Local Interactions (291-299) for the HSE Field 
Device Access (FDA) Agent 290 are shown. The operation of the 
HSE FDA Agent will now be described in terms of these local 
interactions . 

One of the main functions of the HSE FDA Agent 290 is to map 
services already defined for Foundation™ fieldbus System 
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B: HSE Core 



B.l: Network Management Agent 
Virtual Field Device 



B.2: HSE Field Device Access Agent 




Management (SM) (See FF-880 listed in the Reference Set 1 of 
Appendix A herein) and Fieldbus Message Specification (FMS) (See 
FF-870 listed in the Reference Set 1 of Appendix A herein) to and 
from the standard, COTS Ethernet/Internet 280 component. 
5 Generally, the HSE FDA Agent 290 emulates the mapping 

defined by the Foundation™ fieldbus Fieldbus Access Sublayer 
specification (See FF-875 listed in the Reference Set 1 of 
Appendix A herein) . The HSE FDA Agent 290 provides the common 
interface that enables remote applications to access devices of 

10 any type on both the HI Networks 150 and the HSE Network 140. 

Thus the HSE FDA Agent 290 in accordance with the principles 
of the present invention allows systems to be constructed where 
the control is distributed in into various HSE Devices and/or HI 
Devices, and any combinations thereof, as needed by the 

15 particular end user application. 

B.2.1: HSE FDA Agent Local Interfaces 

£3 B. 2. 1(a): Local Interface 291: TCP - The TCP local interface 

jSi 20 291 allows the HSE FDA Agent 290 to send and/or receive FMS 
yft messages using TCP 283. TCP 283 provides interfaces modeled as 

"5 sockets through which the HSE FDA Agent 290 submits a buffer that 

contains one or more messages. 
ff 5 B. 2. 1(b): Local Interface 292: UDP - The UDP local interface 

25 292 allows the HSE FDA Agent 290 to send and/or receive SM 
messages and certain FMS messages using UDP 284. UDP 284 
* provides interfaces modeled as sockets through which the HSE FDA 

0 ; Agent 290 submits a buffer that contains one or more messages. 

B. 2. 1(c): Local Interface 293: HSE NMIB - The HSE FDA Agent 
P~ 30 290 provides a local interface to the HSE NMIB 213 in NMA VFD 
210. The HSE FDA Agent is capable of providing configuration and 
read-only access to NMA VFD 210 via the HSE NMIB Local Interface 
293. 

B. 2. 1(d): Local Interface 294: HSE SMIB - The HSE FDA Agent 
35 290 provides a local interface to the HSE SMIB 212 in NMA VFD 
210. The HSE FDA Agent 290 is capable of providing configuration 
and read-only access to NMA VFD 210 via the HSE SMIB Local 
Interface 294. 

B. 2. 1(e): Local Interface 295: HSE SMK - The HSE FDA Agent 
40 290 conveys HSE SM services to and from the HSE SMK 220 through 
the HSE SMK local interface 295. In accordance with an embodiment 
of the present invention, in a HSE Linking Device, the HSE SMK 
220 communicates locally with each of the Hi interfaces 240, and 
does not use the HSE FDA Agent 290. 
45 B. 2. 1(f): Local Interface 296: HSE LRE - The HSE FDA Agent 

290 maintains a local interface with the HSE LAN Redundancy 
Entity (HSE LRE) 230 of the device through the HSE LRE local 
interface 296. Use of the HSE LRE local interface 296 will be 
described in more detail later. 
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B. 2. 1(g): Local Interface 297: HI Interface - Only HSE FDA 
Agents 290 of a HSE Linking Device interact with the HI 
Interface (s) 240 to access HI Networks 150. The HI local 
interface provides the HSE FDA Agent with FMS and SM access 
through the HSE SMK 220. 

The HSE FDA Agent forwards FMS requests and responses 
received from the TCP Interaction 291 and UDP Interaction 292 to 
HI Network 150 through the HI Interface (s) 240. The HSE FDA 
Agent also forwards HI requests and responses received from a HI 
Network through the Hi Interface Interaction 297 to the Ethernet 
Network 140 using TCP Interaction 291 and UDP Interaction 292. 

Thus, the HSE FDA Agent 290 interacts with the services in 
the HI Network in the same manner as any other application 
program would normally interact with the HI network. 

B. 2. 1(h): Local Interface 298: FBAP VFD - The HSE FDA Agent 
290 uses the FBAP VFD local interface 298 to access the FBAP VFD 
260. Both FMS and SM messages are communicated using the FBAP 
VFD local interface 298. 

B.2.1(i): Local Interface 299: HSE Management Agent - The 
HSE FDA Agent 290 maintains the HSE Management Agent local 
interface 299 with the HSE Management Agent 270 to access certain 
Quality of Service parameters associated with its UDP/TCP 
connections. The use of these parameters by the HSE FDA Agent 
290 is local to the specific UDP/TCP implementation. 

B . 2 . 2 : HSE FDA Agent Operation 

Again referring to FIG. 4, during the configuration of the 
system, HSE SMK 220 uses the local interface 295 for adding HSE 
and/or HI devices to, and deleting the same from, the distributed 
system. An exchange of SM messages is used to identify new (or to 
be deleted) HSE and/or HI Devices in the system. 

For example, after a new HSE Device receives an Internet 
Protocol (IP) address, the new HSE Device periodically announces 
its presence on the Ethernet network 140. HSE Linking Devices 
also announce changes detected on their Hi Network 150. In a 
similar way, HSE SMK uses the local interface 295 to determine 
the location of the function block "tags" that might exist in the 
HSE Devices and/or Hi Devices. 

During operation of the system, the data acquisition, 
display and supervisory control functions, which are typically 
executing on a Personal Computer (PC) connected to the Ethernet 
Network 140, will need to access the data in a HSE Device, a HSE 
Linking Device and/or HI devices connected to the HI Networks 
150. The data access is typically performed using the 
"Client/Server" and/or the "Publisher/Subscriber" messages. 
These data access methods are well known to those familiar with 
Fieldbus messaging. 

For Client/Server and Publisher/Subscriber messages 
originating or terminating in the HSE Device and/or HSE Linking 
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Device, the HSE FDA Agent 290 sends and receives the Ethernet 
Network 140 messages on the local interface 291, provides the 
appropriate mapping to FMS services as previously described 
above, and uses local interfaces 293, 294, 296, 298, and 299 to 
5 access the HSE NMIB 213, HSE SMIB 212, HSE LRE 230, FBAP VFD(s) 
260 and the HSE Management Agent 270, respectively. HSE SMK 220 
is not accessed because it has its own SM messages as previously 
described . 

For Client/Server, Publisher/Subscriber and/or SM messages 
10 originating or terminating in the HI Network 150, the HSE FDA 
Agent 290 uses local interface 297 to send and/or receive 
messages from HI Interface (s) 240. 

If the messages from the HI network 150 are to/from the 
Ethernet Network 140, and are Client/Server or 
15 Publisher/Subscriber messages, HSE FDA Agent 290 uses the FMS 
mapping and local interface 291. If the HI messages to/from the 
Ethernet Network 140 are SM messages, the HSE FDA Agent uses the 
SM mapping and local interface 292. 

If the messages to/from HI Network 150 are to/from the HSE 
20 Linking Device, and are Client/Server or Publisher/Subscriber 
messages, HSE FDA Agent will use FMS mapping and the appropriate 
local interface (except the local interfaces 291 and 292) . 

If the messages to/from HI Network 150 are to/from the HSE 
Linking Device, and are SM messages, HSE FDA Agent will use SM 
25 mapping and the appropriate local interface (except the local 
interfaces 291 and 292) . 
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B.3: HSE System Management Kernel 

30 Referring again to FIG. 2, the HSE system architecture 

includes a HSE System Management Kernel (SMK) 220 for each HSE 
3 device and/or each HSE linking device. The HSE SMK 220 maintains 

information and a level of coordination that provides an 
integrated network environment for the execution and 
35 interoperation of the FBAP VFD 260. 

As previously discussed, HSE SMK 220 provides for routine 
configuration of certain basic system information prior to device 
operation. For example HSE SMK startup takes a device through a 
set of predefined phases for this purpose. During this procedure 
40 a system configuration device recognizes the presence of the 
device on the network and configures basic information into the 
HSE SMIB 212. Once the device receives its basic configuration 
information, its HSE SMK brings it to an operational state 
without affecting the operation of other devices on the network. 
45 It also enables the HSE FDA Agent 290 for use by other functions 
in the device. 



50 
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B.3.1: HSE SMK System Time Synchronization 
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Now referring to FIG. 5, the HSE Management Agent 270 in HSE 
Linking Device 110 uses SNTP 286 to interact with remote SNTP 
Server 510 in Time Master 500 to synchronize System Time 501' in 
HSE MIB 271' with System Time 501 in the Time Master 500. When 
System Time 501' is synchronized with System Time 501, Sync Flag 
(F) 510 in the HSE MIB is set to true by the standard SNTP 
protocol. The Time Master and the HSE Linking Device are 
interconnected using standard, COTS Ethernet equipment 130. This 
synchronization protocol is defined in IETF RFC 2030. 

At any moment, Local Time 502 in HSE SMIB 212 may or may not 
be synchronized with System Time 501' . In order to coordinate 
execution of function blocks in a distributed system, and to 
provide proper time stamping of function block alarms, Local Time 
502 must be Synchronized with System Time 501' . 

All of the function blocks are synchronized with Start of 
Macrocycle, VX T 0 " 520 in HSE SMIB 212. Each HSE Linking Device and 
HSE Device in the system has the same value for T 0 . A function 
block is executed when HSE SMK 220 locally issues a Function 
Block (FB) Start 221 message for the block. Each FB Start 
message is generated based on an offset from T 0 . 

At the start of the macrocycle, T 0 and the offset for each 
block is based on Local Time 502. Therefore each device must 
adjust their Local Time 502 to equal the System Time 501' for the 
system to function properly. However, because each device has a 
hardware clock oscillator that is not perfect, Local Time 502 
will eventually drift out of synchronization with System Time 
501' . 

Fig. 6 shows the process of correcting for the drift in 
accordance with an embodiment of the present invention. In 
particular, when a macrocycle ends in step 601, the HSE SMK 220 
will test the Sync Flag 510 in HSE MIB 271' in step 602. If F 
510 is not true, the process ends in step 606. 

If, on the other hand, it is determined in step 602 above 
that F 510 is true, HSE SMK 220 computes the offset between Local 
Time 502 and System Time 501' in step 603, and sets the Local 
Time 502 to equal the System Time 501' within a value specified 
in a desired time sync class (See Reference Set 1 of Appendix A 
herein) in step 604. 

Once the Local Time 502 is synchronized, in step 605, the 
start time (T 0 ) 520 (shown in FIG. 5) is aligned with start time 
of other devices. 

The start time alignment will now be described with 
references to FIGs . 7A and 7B. FIG. 7A shows the macrocycle 
offset of a device, e.g., device N, before the time 
synchronization, in which the offset 720 represents the error 
that must be corrected in the HSE Device N. As shown, the HSE 
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Device N now has the correct Local Time, but the start time(T 0 ) 
520' of System Macrocycle 700' is not aligned with other devices 
in the distributed system. 

FIG. 7B shows the macrocycle offset of a device, e.g., 
device N, after the time synchronization. The HSE SMK 220 of the 
Device N delays the start time(T 0 ) 520' of the System Macrocycle 
700' by Offset 720 so that the System Macrocycle begins at the 
same time(T 0 ) 520 as, e.g., System Macrocycle 700 in HSE Device 
1. HSE Device N System Macrocycle is now synchronized with the 
System Time, and the synchronization process ends at step 606 
(shown in FIG. 6) . 



Referring to FIG. 4, each HSE Device and HSE Linking Device 
has a HSE Local Area Network (LAN) Redundancy Entity (HSE LRE) 
230. The HSE LRE provides fault tolerance to single failure 
through the use of redundancy. 

HSE LRE periodically sends and receives Redundancy 
Diagnostic Messages over local interface 296. HSE FDA Agent 290 
maps the Diagnostic messages on local interfaces 291 and 292 (See 
HSE Redundancy Specification FF-593 listed in the Reference Set 2 
of Appendix A herein for the Redundancy Diagnostic Message 
Formats . ) 

The Redundancy Diagnostic Messages are sent concurrently on 
Ethernet Network 140 and Ethernet Network 140' . Each device 
receives the Redundancy Diagnostic Messages on Ethernet Network 
140 and Ethernet Network 140' and constructs a local Network 
Status Table (NST) 231. The NST provides detailed status on the 
condition of every HSE device connected to Ethernet Network 140 
and Ethernet Network 140' . The HSE LRE 230 controls which 
Ethernet Network 140 or 140' the HSE Device will use for message 
transmission . 

With this method, all of the network transmission and device 
switchover decisions are distributed into the HSE Devices and the 
system uses standard, COTS Ethernet equipment. 

FIG. 8 illustrates the general topology supported by the 
redundancy aspect of the present invention. The topology shown 
is only an example, showing one of many possible topologies. Any 
topology may be used as long as behavior of the equipment 
providing Ethernet Networks 140 and 140' is standard, COTS 
Ethernet equipment . 

HSE redundancy supports both Ethernet Network redundancy and 
HSE Linking Device redundancy. 



Referring to FIG. 8, HSE Devices 120' and HSE Linking Device 
Pairs 110' have interfaces to both Ethernet Network 140 and 
Ethernet Network 140' . In this example, Ethernet Network 140 is 



B.4: Local Area Network Redundancy Entity 
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provided by COTS Ethernet equipment 130 and Ethernet Network 140' 
is provided by COTS Ethernet equipment 130' . A single failure of 
any one of the Ethernet Networks or one of the Ethernet 
interfaces on a HSE device would cause the HSE LRE previously 
5 described to force communications on the remaining functional 
network. 

B.4.2: HSE Linking Device Redundancy 

The HSE LRE 230 supports HSE Linking Device redundancy. 
10 Redundant HSE Linking Device Pair 160 comprises primary HSE 
Linking Device 110, and standby HSE Linking Device 110' . HI 
Devices 170 are connected by HI Networks 150 to the Redundant HSE 
Linking Device Pair 160. If primary HSE Linking Device 110 fails, 
standby HSE Linking Device 110' will assume control. A HSE device 
15 120' may be made redundant in the same manner as the HSE linking 
device 110, except in a HSE device HI interface (s) is (are) not 
present . 

The present invention provides the necessary diagnostic 
message format to allow an open and interoperable switch-over of 
^ 20 the redundant high speed Ethernet networks and/or the redundant 
Jp HSE linking devices (or HSE devices) . 

*B ; The redundancy method for backup of each HI Network is 

Gfi described in the '178 application, and by the specifications 

C! : listed in Reference Set 1 of Appendix A herein. 

SJ; 25 As can be appreciated, the distributed control system 

architecture in the foregoing description provides an open, 
Q interoperable solution optimized for integration of distributed 

SHi control systems and other control devices in a high performance 

||J ; backbone, provides an open, interoperable solution that provides 

p 30 system time synchronization suitable for distributed control 
Q applications operable over a high performance backbone, and 

Q: provides an open, interoperable solution that provides a fault 

tolerant high performance backbone as well as fault tolerant 
devices that are connected to the backbone. 
35 The preferred embodiments set forth above are to illustrate 

the invention and are not intended to limit the present 
invention. Additional embodiments and advantages within the 
scope of the claimed invention will be apparent to one of 
ordinary skill in the art. 
40 Moreover, while the invention has been described with 

reference to the exemplary embodiments thereof, those skilled in 
the art will be able to make various modifications to the 
described embodiments of the invention without departing from the 
true spirit and scope of the invention. The terms and 
45 descriptions used herein are set forth by way of illustration 
only and are not meant as limitations. In particular, although 
the method of the present invention has been described by 
examples, the steps of the method may be performed in a different 
order than illustrated or simultaneously. Those skilled in the 
50 art will recognize that these and other variations are possible 
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within the spirit and scope of the invention as defined in the 
following claims and their equivalents. 
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